Distance-applied level and effects emulation for improved lip synchronized performance

ABSTRACT

A system and method for creating more realistic lip syncing to pre-recorded vocal tracks during live performance, music videos, film, television, and the like. During a lip synchronized performance, the system and method simulates signal level, proximity effect, and other parameters normally associated with a live performance. A proximity sensor attached in a fixed relationship with a microphone dynamically detects the distance between a microphone and the vocalist. A control data stream that includes dynamic distance information sensed by the proximity sensor is used to increase the signal level of the pre-recorded vocal track as the sensed distance between the microphone and vocalist decreases and decrease the signal level of the vocal track as the distance between the microphone and vocal track increases.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 16/377,182,filed Apr. 6, 2019. The entire contents of U.S. patent application Ser.No. 16/377,182 are hereby incorporated by reference.

BACKGROUND

The present disclosure relates to audio signal processing including lipsynchronization of vocal performance in pre-recorded music.

Lip synchronization, also known as lip syncing, or lip sync, is thepractice of a performer simulating a vocal performance of pre-recordedaudio. Lip syncing is commonly used in film, television, music videos,as well as live concert performance. Lip syncing is a form ofperformance and musical pantomime. One of the goals of lip syncing is tocreate the illusion that the performer is actually singing.

Modern live performance attempts to create a consistent, pleasing, andentertaining audio and visual experience for the audience. With this inmind, musical performers use lip syncing during live performance to helpproduce a high-quality performance for the audience. During a lip-syncedperformance, the performer may either mimic singing or actually sing,but the vocal performance transmitted to the audience is pre-recorded.The pre-recorded acoustic instrument or vocal tracks are known asbacking tracks or pre-recorded backing tracks. Pre-recorded backingtracks can be used to either replace or supplement portions of a liveperformance. Some performances may use a combination of lip sync andlive performance. For example, much of the song or performance may belive, but more vocally challenging portions of the performance may belip-synced to a pre-recorded backing track. In this example, the livesound engineer would mix in the live vocals during most of the song andfade out the pre-recorded vocal backing track. During the more vocallychallenging parts of the performance, the sound engineer will mix in thepre-recorded vocal backing track while fading out the live vocalperformance.

SUMMARY

The inventor runs a performance technology company that produces andarranges backing tracks for well-known musical acts. The inventorobserved that during lip synchronized performances, the desired level ofrealism was not being accomplished because the sound level of thepre-recorded backing track did not correspond to the position of themicrophone with respect to the performer. For example, a performer maygrab the microphone stand and rock it back and forth or move it with thebeat of the music. Alternatively, the performer may step back from ortoward a stand-mounted microphone. He or she may also move a handheldmicrophone toward or away from their mouth. Under those circumstances,the sound level of the singer's voice would not match the sound level ofthe pre-recorded vocal backing track.

The inventor discovered that he could add realism to a lip-syncedperformance by automatically adjusting the sound level of thepre-recorded vocal backing track, in real-time, inversely with thedistance of the vocalist from the microphone. He accomplished this byusing a proximity sensor to dynamically measure the distance between thevocalist and the microphone in real-time. Using data from the proximitysensor, the sound level of the pre-recorded vocal backing trackdynamically decreases as the microphone moves away from the vocalist.The sound level of the pre-recorded vocal backing track dynamicallyincreases as the microphone moves closer to the vocalist. In addition,the pre-recorded vocal backing track can be turned off when theproximity sensor detects that the vocalist is not present.

The proximity sensor is secured a fixed distance from a reference pointon the microphone. For example, the proximity sensor can be secured tothe microphone or the microphone clip a fixed distance from the end ofthe microphone grille. A control data stream that includes dynamicdistance information sensed by the proximity sensor is used to increasethe signal level of the pre-recorded vocal track as the distance betweenthe microphone and vocalist decreases. Likewise, the dynamic distanceinformation sensed by the proximity sensor is used to decrease thesignal level of the vocal track as the distance between the microphoneand vocal track increases. The control data stream can be a serial datastream or other standard computer protocol. The control data stream canbe transmitted wired or wirelessly. For example, by Bluetooth, 802.11,Z-wave, BLE, ZigBee, or other wireless data communication capable oftransmitting real-time data. Wired protocols could include USB,ThunderBolt, FireWire, Ethernet, or Musical Instrument Digital Interface(MIDI). The data stream could be transmitted over standard audio controlprotocols such as MIDI, or Open Sound Control (OSC). It could betransmitted using a proprietary protocol, or could be transmitted simplyas a serial data stream.

The control data stream is transmitted to an audio processing device.The audio processing device may be within, for example, a dedicated box,a digital audio workstation or a digital audio mixing console. Thedigital audio workstation may be a dedicated device or implemented viasoftware within a computer or mobile device. The proximity sensor may beintegrated within a microphone wireless transmission system. Themicrophone wireless receiver can receive the dynamic position data andeither process it internally or send the data out to a computer viastandard computer interface such as USB, Wi-Fi, or Ethernet.

The inventor also noted that the lip-synchronized performance could bemade more realistic by adjusting the frequency response of the vocalbacking track to simulate proximity effect. Proximity effect is anexaggerated low frequency response, for example a booming sound, thatoccurs as the vocalist moves close to the microphone. Proximity effectoccurs on microphones typically used for live performance known ascardioid or hyper-cardioid microphone. These microphones are popular forlive performance because they tend to pick up sound mostly from in frontof the microphone and reject unwanted sound from behind the microphone.

The inventor discovered that he could simulate distance-dependentproximity effect to add realism to a lip-synchronized performance byautomatically adjusting the frequency response of the pre-recorded vocalbacking track, in real-time, inversely with the distance of the vocalistfrom the microphone. The frequencies typically associated with proximityeffect would dynamically adjust as the microphone moved closer orfurther away from the vocalist. In addition, the inventor envisions thatdistortion effects (i.e., distance-dependent microphone distortion),typical of overloading a microphone, could be simulated by dynamicallyusing the proximity sensor. For example, when the vocalist is too closeto the microphone and singing too loudly, the proximity sensor cangenerated dynamic distance data that can activate a feedback effectsemulation module or a distortion effects emulation module to apply theseeffects to the pre-recorded vocal backing track. While proximity effect,distortion effects, and feedback effects are typically undesirable, aselected use of them can add realism to a lip-synced performance. Theinventor believes that part of his contribution to the art is tosimulate effects that are typically undesirable, but nonetheless presentin a live performance, to create realism with a lip-synchronizedperformance.

The inventor envisions that the above described distance simulated soundlevel, proximity effect, distortion effect, and feedback effects (i.e.,simulated microphone feedback) can also be applied to acousticinstruments that are “lip-synced” or mimed to a pre-recorded audio backtrack. For example, a saxophone or trumpet player may be lip syncing hisor her performance. While performing they may be improvising movementsand changing their distance in relationship to the microphone.

This Summary introduces a selection of concepts in simplified form thatare described the Description. The Summary is not intended to identifyessential features or limit the scope of the claimed subject matter.

DRAWINGS

FIG. 1 illustrates a vocalist holding a handheld microphone with aproximity sensor mounted to the microphone where the microphone is adistance D1 from his mouth.

FIG. 2 illustrates the vocalist, microphone, and proximity sensor ofFIG. 1 with the vocalist holding the microphone at a distance D2 fromhis mouth.

FIG. 3 illustrates a vocalist singing at a distance D3 away from themicrophone, where the microphone and a proximity sensor is mounted to amicrophone stand.

FIG. 4 illustrates the vocalist, microphone, proximity sensor, andmicrophone stand of FIG. 3 singing at a distance D4 away from themicrophone.

FIG. 5 illustrates a musician with the sound-producing end of hismusical instrument a distance D5 from a microphone where the microphoneand a proximity sensor are mounted to a microphone stand.

FIG. 6 illustrates the musician, microphone, microphone stand, andproximity sensor of FIG. 5 with the sound-producing end of his musicalinstrument a distance D6 from the microphone.

FIG. 7 illustrates a simplified block diagram of a real-timelevel-versus-distance emulation system applied to a pre-recorded audiobacking track.

FIG. 8 illustrates a first graph demonstrating a first set of examplesof how distance versus level emulation can be applied to the recordedaudio track.

FIG. 9 illustrates a second graph demonstrating a second example of howdistance versus level emulation can be applied to the recorded audiotrack.

FIG. 10 illustrates a simplified flow chart of FIG. 7.

FIG. 11 illustrates a typical graphical user interface with distanceversus level emulator section shown as an audio plugin or module.

FIG. 12 illustrates the graphical user interface of the distance versuslevel emulator section of FIG. 11.

FIG. 13 illustrates a series of microphone frequency response curves,for a typical microphone, each at a given distance to demonstrateproximity effect.

FIG. 14 illustrates combined proximity effect and level-versus-distanceemulation at a distance X1 between the vocalist and the microphone.

FIG. 15 illustrates combined proximity effect and level-versus-distanceemulation at a distance X2 between the vocalist and the microphone.

FIG. 16 illustrates combined proximity effect and level-versus-distanceemulation at a distance X3 between the vocalist and the microphone.

FIG. 17 illustrates combined proximity effect and level-versus-distanceemulation at a distance X4 between the vocalist and the microphone.

FIG. 18 illustrates a simplified block diagram of the distance-appliedlevel, proximity effect, distortion effect, and feedback effectemulation.

FIG. 19 illustrates an alternative simplified block diagram of thedistance-applied level, proximity effect, distortion effect, andfeedback effect emulation.

FIG. 20 illustrates a simplified flow chart for FIG. 18 or 19.

FIG. 21 illustrates a typical graphical user interface for the proximityeffect emulator module of FIG. 18 or 19 shown as an audio plugin.

FIG. 22 illustrates a typical graphical user interface for thedistortion and feedback effects emulation modules of FIG. 18 or 19 shownas an audio plugin or module.

FIG. 23A illustrates a simplified block diagram of a first portion ofthe proximity emulation module of FIG. 18 or 19.

FIG. 23B illustrates a simplified block diagram of a second portion ofthe proximity emulation module of FIG. 18 or 19.

FIG. 24 illustrates a simplified block diagram of the feedback effectemulation module of FIG. 18 or 19.

FIG. 25 illustrates a simplified block diagram of the distortion effectemulation module of FIG. 18 or 19.

FIG. 26 illustrates a microphone and microphone stand where theproximity sensor assembly is mounted on the body of the microphone.

FIG. 27 illustrates a microphone and microphone stand where theproximity sensor assembly is mounted on the microphone clip.

FIG. 28 illustrates a microphone and microphone stand where theproximity sensor assembly is mounted partially within the body of themicrophone with the proximity sensor extending out of the microphonebody and upward around the bottom half of the windscreen.

FIG. 29 illustrates a partially exploded view of the microphone andproximity sensor assembly of FIG. 27 with the hidden elementsillustrated in dashed lines.

FIG. 30 illustrates a simplified electrical block diagram of theproximity sensor assembly where the proximity data is transmitted viaUSB or other wired computer protocols.

FIG. 31 illustrates a simplified electrical block diagram of theproximity sensor assembly where the proximity data is transmitted via awireless antenna.

FIG. 32 illustrates a simplified electrical block diagram of theproximity sensor assembly and microphone where the proximity data istransmitted from the microphone using standard audio control protocolssuch as MIDI or OSC.

FIG. 33 illustrates a simplified system block diagram where thedistance-applied level and effects emulation is processed within adigital audio workstation.

FIG. 34 illustrates a simplified system block diagram where thedistance-applied level and effects emulation is processed within astandalone unit.

FIG. 35 illustrates a simplified system block diagram where thedistance-applied level and effects emulation is processed within adigital audio workstation that receives the proximity sensor data via awireless microphone receiver.

FIG. 36 illustrates an alternative simplified block diagram of thedistance-applied level, proximity effect, distortion effect, andfeedback effect emulation that includes user controls on the microphone.

FIG. 37 illustrates a microphone with user controls on the microphone.

FIG. 38 illustrates a microphone with an alternative set of usercontrols on the microphone.

DESCRIPTION

The terms “left,” “right,” “top, “bottom,” “upper,” “lower,” “front,”“back,” and “side,” are relative terms used throughout the Descriptionto help the reader understand the figures. Unless otherwise indicated,these do not denote absolute direction or orientation and do not imply aparticular preference. When describing the figures, the terms “top,”“bottom,” “front,” “rear,” and “side,” are from the perspective of theaudience located front and center of the performer. Specific dimensionsare intended to help the reader understand the scale and advantage ofthe disclosed material. Dimensions given are typical and the claimedinvention is not limited by the recited dimensions.

The following terms are used throughout this disclosure and are definedhere for clarity and convenience.

Lip Syncing: As defined in this disclosure, lip syncing or lipsynchronization means miming or mimicking to a pre-recorded performanceusing a microphone or a prop microphone that is visible to the audience.As defined in this disclosure, lip syncing or lip synchronization is notlimited to miming or mimicking pre-recorded vocal performances, but mayalso include miming or mimicking pre-recorded acoustic instrument wherethe microphone or prop microphone associated with that acoustic musicalinstrument is visible to the audience.

Stage Microphone: Throughout this disclosure, stage microphone meansrefers to a microphone or a prop microphone that is identifiable to anaudience member or observer as a microphone.

Prop Microphone: Throughout this disclosure a “prop microphone” or a“microphone prop” is a device that mimics the appearance of a microphonebut is not capable of functioning as a microphone or is a functioningmicrophone with the sound producing capabilities disabled.

Non-transitory computer-readable medium: As used in this disclosure, theterm “non-transitory”, means that the medium is tangible and not asignal. The term “non-transitory computer readable medium” encompassesstorage devices that do not necessarily store information permanently,for example, random access memory (RAM). Program instructions and datastored on a non-transitory computer-readable storage medium may betransmitted by transmission media or electrical signals. These signalsmay be conveyed by wires, signal traces, or wirelessly.

Performer: As defined in this disclosure, a performer refers to avocalist or to an acoustic instrument in combination with the musicianplaying the acoustic instrument.

Throughout this disclosure the term “sound engineer” refers to a genericuser of the systems and methods described within this disclosure.

As noted in the Background, modern live performance attempts to create aconsistent, pleasing, and entertaining audio and visual experience forthe audience. With this in mind, vocalists use lip syncing during liveperformance to help produce a high-quality performance for the audience.During a lip-synced performance, the performer may either mimic singingor actually sing, but the vocal performance transmitted to the audienceis pre-recorded. The pre-recorded music or vocal tracks are known asbacking tracks, or pre-recording audio backing tracks, as they can beused to either replace or supplement portions of a live performance. Asdiscussed in the Summary, the inventor runs a live performancetechnology company that produces and arranges backing tracks forwell-known musical acts. The inventor observed that during lipsynchronized performances, the desired level of realism was not beingachieved because the sound level of the pre-recorded backing track didnot correspond to the position of the microphone with respect to theperformer. Referring to FIGS. 1 and 2, a vocalist 1 may move the stagemicrophone 2 toward their mouth as in FIG. 1, or away from their mouthas in FIG. 2. When the stage microphone 2 is mounted on a microphonestand 4, as in FIGS. 3 and 4, the vocalist 3 may step toward the stagemicrophone 2, as in FIG. 3, or step back from the stage microphone 2, asin FIG. 4. The vocalist movements described for FIGS. 1-4 may not matchthe pre-recorded vocal backing tracks being played to the audience. Thismay cause the audience to perceive that the performance is not live. Theinventor made a similar observation for musicians playing acousticinstruments that are supported or replaced by a pre-recorded backingtrack of their instruments. Referring to FIG. 5, the musician 6 mighthold the sound-producing end 5 a of his acoustic instrument 5 close tothe stage microphone 2 and microphone stand 4, shown here as distanceD5. Referring to FIG. 6, the musician 6 may step back from the stagemicrophone 2 and microphone stand 4. If the musician's movementsdescribed for FIGS. 5 and 6 did not match the performance of thepre-recorded backing track, this could also cause the perception amongthe audience that the performance is not live.

Referring to FIGS. 1-6, the inventor discovered that he could compensatefor the movement of vocalists 1, 3 with respect to the stage microphone2, as exemplified in FIGS. 1-4, or the movement of the musicians 6 withrespect to the stage microphone 2, as exemplified in FIGS. 5 and 6, byautomatically adjusting the signal level of the pre-recorded backingtrack, in real-time, inversely with the distance of the vocalists 1, 3or musician 6 from the microphone. The distance can be measured using aproximity sensor 7 a. The signal level would dynamically decrease as themicrophone is moved away from the vocalist or dynamically increase asthe microphone moves closer to the vocalist. For example, in FIGS. 1 and2, the signal level of the pre-recorded vocal backing track woulddynamically decrease as the stage microphone 2 is moved from distance D1in FIG. 1 to distance D2 in FIG. 2, from distance D3 in FIG. 3 todistance D4 in FIG. 4, and from distance D5 in FIG. 5 to distance D6 inFIG. 6. Distances D1, D2, D3, D4 being the distance from the end 2 a ofthe microphone grille 2 b to the mouth of the vocalist 1, 3. DistancesD5, D6 being the distance between the end 2 a of the microphone grille 2b to the sound-producing end 5 a of the acoustic instrument 5. Forexample, in FIGS. 5 and 6, the sound-producing end 5 a is the bell ofthe trumpet. In addition, the pre-recorded vocal backing track can beturned off when the proximity sensor 7 a detects that the vocalist isnot present.

The inventor discovered, through experimentation, that in order torealistically track the movement of the vocalist 1 (FIGS. 1, 2) andvocalist 3 (FIGS. 3 and 4) or a musician 6 (FIGS. 5 and 6), theproximity sensor must have sufficient sensitivity and speed. Theinventor discovered that time-of-flight sensors are generallysatisfactory for use as a proximity sensor 7 a within the meaning ofthis disclosure. Time-of-flight sensors typically have sampling periodsof 20 milliseconds or less. An example of a satisfactory time-of-flightsensor for use as a proximity sensor 7 a is the VL53L0X by STMicroelectronics. The inventor found that ultrasonic sensors withsampling periods of 250 milliseconds or greater were not satisfactory.Time-of-flight sensors measure distance to an object based on the timeit takes for light emitted by the sensor to be reflected back from theobject. Time-of-flight sensors typically use one or more infrared lasersto measure time-of-flight.

Continuing to refer to FIGS. 1-6, the proximity sensor 7 a is housedwithin a proximity sensor assembly 7. The proximity sensor assembly 7 isattached in a fixed relationship with the stage microphone 2. That is,it maintains a fixed distance from a reference point on the stagemicrophone 2 such as the end 2 a of the microphone grille 2 b. In FIGS.1 and 2, the proximity sensor assembly 7 is shown positioned on themicrophone body 2 c. The microphone in FIGS. 1 and 2 is a wirelessmicrophone with a wireless transmitter 2 d projecting out of the end ofthe microphone body 2 c. In FIGS. 3-6, the proximity sensor assembly 7is attached to the microphone clip 4 a. The microphone clip 4 a, ormicrophone mount, is a mechanical device that attaches the stagemicrophone 2 to the microphone stand 4. Typically, microphone clips 4 aallow the microphone to swivel in relation to the microphone stand.

In FIGS. 1-6, and throughout this disclosure, for a purelylip-synchronized performance, i.e., where the stage microphone 2 is notbeing used to transmit a live audio signal, the stage microphone 2 canbe a “prop microphone,” or “dummy microphone.”

FIGS. 7-12 illustrates an example of how the system works. Referring toFIG. 7, the audio data stream 8 of the pre-recorded audio backing track9 feeds a level-versus-distance emulation module 10. Thelevel-versus-distance emulation module 10 adjusts the signal level ofthe audio data stream 8 inversely with distance using control signal 11to provide distance information. The control signal 11 is based on thedynamic distance data stream 12 generated by the proximity sensor 7 a.With the proximity sensor 7 a mounted in a fixed distance relationshipwith respect to the stage microphone 2 of FIGS. 1-6, the dynamicdistance data stream 12 is generated by movement of the stage microphone2 with respect to the vocalist 1, 3, as in FIGS. 1-4, or movement of thestage microphone 2 with respect to the sound-producing end 5 a of theacoustic instrument 5, as in FIGS. 5 and 6. In some embodiments, thelevel-versus-distance emulation module 10 can be controlled directly bythe dynamic distance data stream 12. In other embodiments, such as adigital audio workstation, the dynamic distance data stream 12 may needto be first converted to an audio industry control protocol, such asMIDI or OSC. In FIG. 7, the dynamic distance data stream 12 is convertedto the control signal 11 by the control signal protocol conversionmodule 13. The control signal protocol conversion module generates acontrol signal 11 such as the above-mentioned audio industry controlprotocol or other standard control protocol.

The level-versus-distance emulation module 10 uses the dynamic distancedata stream 12 provided by the control signal 11 to adjust the audiodata stream 8 of the pre-recorded audio backing track 9 inversely withdistance. The level-versus-distance emulation module 10 typicallyadjusts the audio data stream 8 according to the inverse square law.That is, for each doubling of distance, the signal level falls by halfor by −6 dB and for each halving of distance, the signal level increasesby 6 dB. This emulates what the audience would likely hear if they werelistening to a live performance. This is because microphone signallevels decrease with the square of the distance that the sound source isfrom the stage microphone 2. Referring to FIGS. 8 and 9, this isrepresented by level-versus-distance curve 14 a illustrated as a solidlevel-versus-distance curve. FIGS. 8 and 9 represent typicallevel-versus-distance adjustment laws followed by thelevel-versus-distance emulation module 10 of FIG. 7. The vertical scaleshows relative signal levels in decibels (dB), with the top of the scalebeing 0 dB. The horizontal scale shows distance in meters.

This process of dynamically adjusting the signal level of a pre-recordedaudio signal according to distance between a stage microphone 2 andacoustic instrument 5 (FIGS. 5 and 6) or vocalist 1, 3 (FIGS. 1-4) canbe summarized by the flow chart 15 of FIG. 10. Referring to FIGS. 1-7and 10, in step 16 (FIG. 10), the proximity sensor 7 a (FIG. 7)dynamically detects distance to acoustic instrument 5 (FIGS. 5 and 6) orvocalist 1, 3 (FIGS. 1-4). In step 17 (FIG. 10), the dynamic distancedata stream 12 (FIG. 7) is converted to the control signal 11 (FIG. 7).In step 18 (FIG. 10), the signal level of the audio data stream 8 of thepre-recorded audio backing track 9 (FIG. 7) is adjusted inversely withdistance to emulate distance between the stage microphone 2 and acousticinstrument 5 (FIGS. 5 and 6) or vocalist 1, 3 (FIGS. 1-4). The processis continuously repeated as illustrated in FIG. 10.

Referring to FIG. 7, it may be desirable to provide a mixture of theemulated output 19 from the level-versus-distance emulation module 10and the audio data stream 8. This is known as a wet/dry mix. Tofacilitate this, the audio data stream 8 and the emulated output 19 aresummed 20 or mixed, creating the audio output 21. The ratio of emulatedoutput 19 (i.e., “wet output”) to the audio data stream 8 (i.e., “dryoutput”) can be controlled by the sound engineer by user parametercontrols 22 via control signal path 23.

The level-versus-distance emulation module 10, the control signalprotocol conversion module, as associated summing and mixing can beimplemented by a processor, or more than one processor, executinginstructions stored a non-transitory computer readable medium such asROM, RAM, or FLASH memory, or memory embedded in the processor. Theprocessor can be a digital signal processor (DSP), an FPGA, a PLD, ASIC,a microprocessor, or a microcontroller, or any other processor capableof executing the instructions and performing the functions described.The pre-recorded audio backing track 9 can be stored and transmitted tothe processor from memory such as a hard drive, flash memory, DVD, adedicated digital audio storage unit, or transmitted over a network froma non-transitory computer readable medium. Alternatively, one or more ofthese elements can be implemented in dedicated hardware.

Referring to FIGS. 7-9, the user parameter controls 22 (FIG. 7) can alsocontrol the behavior of the level-versus-distance emulation module viacontrol signal path 24. For example, the level-versus-distance curve canbe scaled as illustrated by level-versus-distance curves 14 b, 14 c inFIG. 8. Level-versus-distance curves 14 a, 14 b, 14 c all follow theinverse square law, but level-versus-distance curves 14 b, 14 c areattenuated −6 dB and −12 dB, respectively as compared withlevel-versus-distance curve 14 a. This may be helpful for setting the 0dB based on the habits of the performer. For example, some vocalist 1like to habitually hold the microphone close to their lips, as shown inFIG. 1, while others, like the vocalist 3 in FIG. 3 do not. The soundengineer may also find it helpful to change the slope or even the shapeof the distance versus level curve. For example, in FIG. 9, the userparameter controls 22 of FIG. 7 is used to adjust the slope of thelevel-versus-distance curve 14 d, 14 e. Level-versus-distance curve 14 dhas a shallower slope than level-versus-distance curve 14 a (i.e.,shallower than the inverse square law). Level-versus-distance curve 14 ehas a steeper slope than level-versus-distance curve 14 a (i.e., steeperthan the inverse square law).

Referring to FIG. 7, the user parameter controls 22 can be implementedas hardware controls. For example, rotary or linear potentiometers,rotary or linear encoders, switches, as well as other controls typicallyfound on audio control interfaces. The user parameter controls 22 can beimplemented as soft controls on a graphical user interface such assoftware controls on a digital audio workstation graphical userinterface. The user parameter controls 22 can be implemented as acombination of hardware controls and soft controls. For example, adigital audio workstation with both a graphical user interface and ahardware control surface. FIG. 11 shows a simplified view of graphicaluser interface 25 of a digital audio workstation. Referring to FIG. 11,the graphical user interface 25 includes various audio tracks and MIDIor OSC control tracks. For example, the audio data stream 8 of thepre-recorded audio backing track 9 and an audio track 26. The audiotrack 26 can be pre-recorded or a live audio stream. Both tracks arerepresented by audio waveforms. Below audio track 26 is the plugin area27. In this example, the dynamic distance control plugin 28 includes thelevel-versus-distance emulation module 10 and user parameter controls22, both from FIG. 7. The dynamic distance control plugin 28 receivesthe control signal 11 from FIG. 7 as control track 29. Control track 29,in this example, would be MIDI track containing MIDI information thatrepresents the distance data received from the proximity sensor 7 a ofFIG. 7. For a digital audio workstation, the control track 29 istypically MIDI, OSC, or the program's proprietary API. The dynamicdistance control plugin 28 applies dynamic level control to the audiodata stream 8 using the level-versus-distance emulation module 10 ofFIG. 7. The dynamic distance control plugin 28 also can include the userparameter controls 22 of FIG. 7.

FIG. 12 illustrates the user interface of the dynamic distance controlplugin 28 of FIG. 11 in more detail. Referring to FIG. 12, the dynamicdistance control plugin 28 is illustrated with a sensor display portion30 and user controls 31. The sensor display portion 30 includes a sensorreading 30 a that dynamically displays the distance the proximity sensor7 a of FIGS. 1-7 is from the vocalist 1, 3 of FIGS. 1-4 or thesound-producing end 5 a of the acoustic instrument 5 of FIGS. 5 and 6.Along with the sensor reading 30 a is a sensor distance graphic 30 bthat dynamically shows the distance the vocalist or instrument is fromthe microphone. The sensor display portion 30 also includes an out ofrange indicator 30 c that indicates that the object (i.e. vocalist oracoustic instrument) are out of range. The user controls 31 can includea DAW selector 31 a, a scale factor 31 b, slope control 31 c, minimumdistance control 31 d, maximum distance control 31 e, offset distancecontrol 31 f, and destination control 31 g. The DAW selector 31 a thatallows the sound engineer to select which digital audio workstation toapply the dynamic distance control plugin 28. Here, the digital audioworkstation is selected via a menu 31 h. The scale factor 31 b can scaledown the maximum signal level similar to level-versus-distance curves 14b, 14 c of FIG. 8. The slope control 31 c can adjust thelevel-versus-distance law. The default would typically be the inversesquare law. This control can make the level-versus-distance lawshallower or steeper in a similar manner as illustrated forlevel-versus-distance curves 14 d, 14 e of FIG. 9. The minimum distancecontrol 31 d, and maximum distance control 31 e set the minimum andmaximum distance detectable from the proximity sensor 7 a of FIGS. 1-7,respectively. The offset distance control 31 f of FIG. 12 compensatesfor the distance the proximity sensor 7 a is away from the end 2 a ofthe microphone grille 2 b also in FIGS. 1-7. For example, the offsetdistance control 31 f of FIG. 12 can compensate for offset distance 51in FIGS. 1 and 2 or offset distance S2 in FIGS. 3-6. Offset distances51, S2 being the distance between the end 2 a of the microphone grille 2b and the proximity sensor 7 a. Referring to FIG. 12, the destinationcontrol 31 g selects which track the dynamic distance control plugin 28will be applied to. For most Digital Audio Workstation software, thedestination control 31 g will be mapped directly to the pre-recordedaudio backing track 9. In some software it may be necessary to map thedestination control 31 g to a MIDI or instrument track to host thedynamic distance control plugin.

As discussed in the Summary, the inventor also noted that theperformance could be made more realistic by adjusting the frequencyresponse of the vocal backing track to simulate proximity effect as thevocalist moves closer to a typical stage microphone. FIG. 13 illustratesa frequency versus level graph 32 for a typical cardioid microphonewhere each line illustrates the frequency response for a source at agiven distance from the sound source. Frequency response curve 32 arepresents a distance between the microphone and sound source of 0.005meters (0.125 inches). Frequency response curve 32 b represents adistance between the microphone and sound source of 0.025 meters (0.984inches). Frequency response curve 32 c represents a distance between themicrophone and sound source of 0.05 meters (2.00 inches). Frequencyresponse curve 32 a represents a distance between the microphone andsound source of 0.5 meters (23.6 inches). As shown by the frequencyversus level graph 32, at close distances, frequencies between 100 Hertz(Hz.) and 300 Hz. can be significantly boosted. For example, infrequency response curve 32 a the signal is boosted by approximately12.5 dB at 200 Hz. In contrast, in frequency response curve 32 d, thesignal is attenuated by approximately −2.5 dB at 200 Hz. The graphfrequency response curve portion 32 e represented by the solid line,shows that at approximately 1.2 kHz. the frequency response of themicrophone is unaffected by distance from the sound source.

The inventor discovered that he could simulate proximity effect for animprovising lip-syncing vocalist by automatically adjusting the lowfrequency response of the pre-recorded audio backing track, inreal-time, inversely with the distance of the vocalist from themicrophone to mimic or approximate proximity effect. This could be donein combination with the level-versus-distance compensation described forFIGS. 7-12. Referring to FIGS. 14-18, a proximity effect emulationmodule 33 (FIG. 18) dynamically increases the low frequency response ofthe audio data stream 8 of the pre-recorded audio backing track 9 (FIG.18) as the stage microphone 2 (FIGS. 14-17) is moved toward the vocalist34 (FIGS. 14-17) or dynamically decreases the low frequency response asthe stage microphone 2 moves away from the vocalist 34. In addition,other frequency response changes that occur in response to distance canalso be emulated. At the same time, the level-versus-distance emulationmodule 10 (FIG. 18) adjusts the signal level of the output of the summer51 based on the distance between the proximity sensor 7 a (FIG. 18) andthe vocalist 34.

FIGS. 14-17 illustrate a frequency response graph 35 and relative signallevel 36, and distance between the stage microphone 2 and vocalist 34.Referring to FIGS. 14-17, for the sake of example, we can assume thesame distances used in FIG. 13. That is, X1=0.005 meters (FIG. 14),X2=0.025 meters (FIG. 15), X3=0.05 (FIG. 16), and X4=0.5 meters (FIG.17). Referring to FIGS. 14 and 18, the proximity effect emulation module33 (FIG. 18) boosts the lower frequency portion of the frequencyresponse curve 35 a (FIG. 14) in a similar manner as the frequencyresponse curve 32 a of FIG. 13. In FIG. 18, the level-versus-distanceemulation module 10 adjusts the signal level of the audio data stream 8of the pre-recorded audio backing track 9 according to the dynamicdistance data stream 12 from the proximity sensor 7 a. For ease ofunderstanding, the relative signal level 36 in FIG. 14 is illustrated as0 dB. Referring to FIGS. 15 and 18, the proximity effect emulationmodule 33 (FIG. 18) boosts the lower frequency portion of the frequencyresponse curve 35 b (FIG. 15) in a similar manner as the frequencyresponse curve 32 b of FIG. 13. The level-versus-distance emulationmodule 10 decreases the relative signal level 36 to −18 dB in FIG. 15 asthe proximity sensor 7 a detects that the distance between the stagemicrophone 2 and the vocalist 34 quadrupled. Referring to FIGS. 16 and18, the proximity effect emulation module 33 (FIG. 18) adjusts the lowerfrequency portion of the frequency response curve 35 c (FIG. 16) in asimilar manner as the frequency response curve 32 c of FIG. 13. Thelevel-versus-distance emulation module 10 decreases the relative signallevel 36 from −18 dB in FIG. 15 to −24 dB in FIG. 16 as the proximitysensor 7 a detects that the distance between the stage microphone 2 andthe vocalist 34 doubled from FIG. 15. Referring to FIGS. 17 and 18, theproximity effect emulation module 33 (FIG. 18) adjusts the lowerfrequency portion of the frequency response curve 35 d (FIG. 17) in asimilar manner as the frequency response curve 32 d of FIG. 13. Thelevel-versus-distance emulation module 10 decreases the relative signallevel 36 from −24 dB in FIG. 16 to −45 dB in FIG. 17 as the proximitysensor 7 a detects that the distance between the stage microphone 2 andthe vocalist 34 has decreased by nearly twelve times from FIG. 16.

As stated in the Summary, the inventor envisions that distortioneffects, typical of overloading a microphone could be simulated bydynamically detecting the distance between the microphone and thevocalist. For example, when the vocalist is too close to the microphoneand singing too loudly, feedback or distortion can occur. Whiledistortion effects and feedback are typically undesirable, a selecteduse of them along with emulated proximity effect and distance-basedlevel control can add realism to a lip-synced performance. FIG. 18 showsan example of a simplified system block diagram that applies distortioneffect, feedback effect, proximity effect, and distance-based level to apre-recorded audio backing track to add realism to a lip-syncedperformance. FIG. 19 shows an alternative example of a simplified systemblock diagram that applies distance-dependent distortion effect,simulated microphone feedback, distance-dependent proximity effect, anddistance-based level to a pre-recorded audio backing track to addrealism to a lip-synced performance. The primary difference between thetwo figures being the position of the level-versus-distance emulationmodule 10. FIG. 18 also illustrates an optional live audio path. FIG. 20shows a simplified flow chart 37 illustrating a process flow chart forFIG. 18 or 19.

Referring to FIGS. 18-20, in step 45 (FIG. 20), the proximity sensor 7 a(FIGS. 18 and 19) dynamically detects the distance from the vocalist oracoustic instrument. This circuitry associated with the proximity sensor7 a generates the dynamic distance data stream 12 (FIGS. 18 and 19).This circuitry may be integrated within or may be separate from theproximity sensor 7 a. For example, the circuitry may be housed in theproximity sensor assembly 7. In step 46 (FIG. 20), the control signalprotocol conversion module 13 (FIGS. 18 and 19) converts the dynamicdistance data stream 12 to a control signal 11 (FIGS. 18 and 19). Thecontrol signal can be a standard musical control protocol, such as MIDIor OSC, or other standard control protocols. In step 47 (FIG. 20), thedynamic distance data stream 12 is applied to the proximity effectemulation module 33 (FIGS. 18 and 19) to dynamically adjust thefrequency response of the audio data stream 8 of the pre-recorded audiobacking track 9 (FIGS. 18 and 19) to emulate distance dependentproximity effect as described in the preceding paragraphs. In step 48(FIG. 20), the dynamic distance data stream 12 is applied to thedistortion effect emulation module 41 (FIGS. 18 and 19) to dynamicallyapply distortion effects to the audio backing track to emulate distancedependent distortion. In step 49 (FIG. 20), the dynamic distance datastream 12 is applied to the feedback effect emulation module 42 (FIGS.18 and 19) by dynamically adding simulated microphone feedback to theaudio data stream 8 of the pre-recorded audio backing track 9 to emulatedistance dependent feedback. In step 50, as illustrated in FIG. 18, thedynamic distance data stream 12 is applied to the level-versus-distanceemulation module 10 to adjust the level of pre-recorded audio backingtrack 9 to emulate distance dependent microphone signal levels. In step50, as illustrated in FIG. 19, the dynamic distance data stream 12 isapplied to the level-versus-distance emulation module 10 to adjust thelevel of mix of the outputs of the proximity effect emulation module 33,the distortion effect emulation module 41 and the feedback effectemulation module 42 to emulate distance dependent microphone signallevels. Referring to FIGS. 18 and 19, the control signal protocolconversion module 13 uses the dynamic distance data stream 12 togenerate a control signal 11 that controls the proximity effectemulation module 33, the distortion effect emulation module 41, thefeedback effect emulation module 42, and the level-versus-distanceemulation module 10. In particular, if the above discussed modules wereimplemented within a digital audio workstation plugin, control signal 11could be MIDI or OSC since these protocols are typically used to controldigital audio workstation plugins. However, in some instances, thecontrol signal protocol conversion module 13 could be bypassed oreliminated and the proximity effect emulation module 33, the distortioneffect emulation module 41, the feedback effect emulation module 42, andthe level-versus-distance emulation module 10 could be controlleddirectly by the dynamic distance data stream 12. For example, customsystems that do not conform to standard digital audio workstationprotocol.

Referring to FIG. 18, the proximity effect emulation module 33, thedistortion effect emulation module 41, and the feedback effect emulationmodule 42 each feed respective level control modules 52, 53, 54. Theoutput of the level control modules 52, 53, 54 are mixed via a summer51. Optionally, the audio data stream 8 of the pre-recorded audiobacking track 9 can be also mixed with the outputs of the level controlmodules 52, 53, 54 in order to create a wet/dry mix between the audiodata stream 8 of the pre-recorded audio backing track 9 and the outputsof the level control modules 52, 53, 54. The output of the summer 51feeds the level-versus-distance emulation module 10. Referring to FIG.19, the proximity effect emulation module 33, the distortion effectemulation module 41, the feedback effect emulation module 42, and thelevel-versus-distance emulation module 10 each feed respective levelcontrol modules 52, 53, 54, 55. The output of the level control modules52, 53, 54, 55 are mixed via a summer 51. As in FIG. 18, in FIG. 19 theaudio data stream 8 of the pre-recorded audio backing track 9 can bealso mixed with the outputs of the level control modules 52, 53, 54, 55in order to create a wet/dry mix between the audio data stream 8 of thepre-recorded audio backing track 9 and the outputs of the level controlmodules 52, 53, 54. In FIG. 19, the audio output 57 that results fromsumming the outputs of the level control modules 52, 53, 54 via thesummer 56 can be sent to a mixing console or similar device, either viadigital-to-analog conversion or directly in the digital domain.

Referring to FIG. 18, it may be desirable to provide the audience with amixture of a live vocal performance along with the audio data stream 8of the pre-recorded audio backing track 9 with distance-emulated effects(i.e., the proximity effect emulation module 33, the distortion effectemulation module 41, feedback effect emulation module 42, andlevel-versus-distance emulation module 10). For example, the liveperformance vocals could be used for all but challenging parts of thevocal performance. In this example, the stage microphone 2 would be areal microphone. During the challenging portions of the vocalperformance, the audio data stream 8 of the pre-recorded audio backingtrack 9 of the vocalist with distance-emulated effects could be mixedin. Similarly, for acoustic instruments, particularly challenging partsof the live performance could be replaced by the audio data stream 8 ofthe pre-recorded audio backing track 9. In order to accommodate theseand similar situations, a live audio signal path 58 can optionally bemixed with the output of the level-versus-distance emulation module 10and summed 59 to create the audio output 57. The live audio signal path58 would typically be fed by the stage microphone 2 associated with theproximity sensor 7 a via a microphone preamplifier and analog-to-digitalconverter 60. The output of the microphone preamplifier andanalog-to-digital converter 60 can directly generate the live audiosignal path 58, or as illustrated, could feed a signal processing module61 before being mixed into the summer 59. The signal processor couldinclude typical vocal processing such as compression, noise gating, orequalization.

Referring to FIGS. 18 and 19, the proximity effect emulation module 33,the distortion effect emulation module 41, the feedback effect emulationmodule 42, the level-versus-distance emulation module 10, the levelcontrol modules 52, 53, 54, 55 and the summers 51, 56, 59 can beaffected by user parameter controls 22 via a control signal 62, severalcontrol signals, or a control signal bus. As previously discussed, theuser parameter controls 22 can be implemented as hardware controls. Forexample, rotary or linear potentiometers, rotary or linear encoders,switches, as well as other controls typically found on audio controlinterfaces. The user parameter controls 22 can be implemented as softcontrols on a graphical user interface such as software controls on adigital audio workstation graphical user interface. The user parametercontrols 22 can be implemented as a combination of hardware controls andsoft controls. For example, a digital audio workstation with both agraphical user interface and a hardware control surface.

The level-versus-distance emulation module 10, the control signalprotocol conversion module 13, the proximity effect emulation module 33,the distortion effect emulation module 41, the feedback effect emulationmodule 42, level control modules 52, 53, 54, 55, and associated summingand mixing can be implemented by a processor, or more than oneprocessor, executing instructions stored a non-transitory computerreadable medium such as ROM, RAM, or FLASH memory, or memory embedded inthe processor. The processor can be a digital signal processor (DSP), anFPGA, a PLD, ASIC, a microprocessor, or a microcontroller, or any otherprocessor capable of executing the instructions and performing thefunctions described. The pre-recorded audio backing track 9 can bestored and transmitted to the processor from memory such as a harddrive, flash memory, DVD, a dedicated digital audio storage unit, ortransmitted over a network from a non-transitory computer readablemedium. Alternatively, one or more of these elements can be implementedin dedicated hardware.

FIG. 21 is typical of a graphical user interface for a proximity effectplugin 38. This illustrates user parameter controls 22 typical for theproximity effect emulation module 33 of FIGS. 18 and 19. Referring toFIG. 21, the sound engineer typically selects a microphone they wish toemulate the proximity effect for from the microphone present panel 63and pop-up panel 63 a. In this case, the selected microphone is a ShureBeta SM58A. The sound engineer can view the frequency response curves onthe frequency response graph 64 for a given distance, indicated bydistance indicator 65 a, by adjusting the slide bar 65 b of theproximity slider control 65. The proximity slider control 65 also caninclude pop-up menus (not shown) associated with a minimum distancecontrol 65 c and a maximum distance control 65 d that allows the soundengineer to set the minimum and the maximum distance limits,respectively, for the proximity effect emulation module (i.e., beyondthese limits no further changes in frequency response will occur). Thesound engineer typically chooses the same microphone that is being usedon stage by the vocalist or musician in order to create an accurateillusion including the proximity effect for the lip-synchronizedperformance. However, the sound engineer is free to choose anymicrophone or a custom preset. If the sound engineer wishes to create acustom preset, for example, the User Preset_01, they can use the grabhandles 64 a, 64 b, 64 c, 64 d, 64 e, 64 f, 64 g, 64 h, 64 i and thegrab bars 64 j, 64 k to create a custom preset for a set of frequenciesresponse curves. In addition, the sound engineer can adjust the wet/dryratio using a wet/dry ratio control 66. The audio data stream 8 of thepre-recorded audio backing track 9 of FIGS. 18 and 19 can be muted at aspecific distance using the mute control panel 67. This can be overridden by a similar control, if present on the dynamic distance controlplugin 28.

FIG. 22 illustrates an effects plugin 40 that implements the userparameter controls 22 typical for the distortion effect emulation module41 and feedback effect emulation module 42. The sound engineer canchoose a preset from the microphone preset panel 68. The preset can beselected via a pop-up menu as described for the proximity effect plugin38 of FIG. 21. The distance versus intensity graphic display 69 willshow the activation curve for the distortion and feedback effects forthe preset. Selecting the distortion button 72 a from the distortioncontrols 72 will display the graph for the distortion effect on thedistance versus intensity graphic display 69. Selecting the feedbackbutton 73 a will display the graph for the feedback effect on thedistance versus intensity graphic display 69. The sound engineer cancustomize the curve by grab handles 69 a, 69 b, 69 c, 69 d and storethis in a custom preset. The minimum and maximum distances can be setusing a minimum distance control 69 e and a maximum distance control 69f, respectively in a similar manner as described for the proximityeffect plugin 38 of FIG. 21. The current distance between the proximitysensor 7 a of FIGS. 1-6, and the vocalist or acoustic instrument isindicated by proximity indicator 71. The distortion controls 72 allowthe sound engineer to set the character of the distortion and theintensity of the distortion with the character control 72 b and theintensity control 72 c. For example, the character control can controlthe nature and intensity of the harmonic overtones present in thedistortion. The feedback controls 73 is illustrated with the probabilitycontrol 73 b and the character control 73 c. The probability control 73b determines the probability that feedback will occur and the charactercontrol 73 c controls the sonic character of the feedback (i.e., lowerfrequency versus high frequency squealing). Both the distortion controls72 and feedback controls 73 include a wet/dry control 72 d, 72 e,respectively, to control the mix between the unmodified signal from theaudio data stream 8 of the pre-recorded audio backing track 9 of FIGS.18 and 19 and the outputs of the distortion effect emulation module 41and the feedback effect emulation module 42 respectively.

FIGS. 23A and 23B show an implementation of the proximity effectemulation module 33 of FIGS. 18 and 19 in a simplified block diagram 39a, 39 b. FIGS. 23A and 23B should be viewed together as a single diagramwith corresponding capital letters A, B, and C surrounded by a circle oneach sheet representing common points or continuations from the previoussheet. Referring to FIG. 23A, based on proximity data from the controlsignal lithe frequency response lookup table 74 adjusts audio filters toshape the frequency response in real-time to create a distance-basedfrequency response curve. Referring to FIGS. 23A and 23B, the controlsignal 74 a generated from the frequency response lookup table 74,controls a low-shelf equalizer 75, band-boost/cut equalizers 76, 77, 78,79, 80, 81, and a high shelf equalizer 82. Each can include a frequencyparameter, a gain parameter, and a bandwidth parameter that iscontrolled by the control signal 74 a from the frequency response lookuptable 74 in response to the proximity data from the control signal 11.For example, the low-shelf equalizer 75 can include a frequencyparameter 75 a, a gain parameter 75 b, and a bandwidth parameter 75 c.The band-boost/cut equalizers 76, 77, 78, 79, 80, 81 can includefrequency parameters 76 a, 77 a, 78 a, 79 a, 80 a, 81 a, a gainparameter 76 b, 77 b, 78 b, 79 b, 80 b, 81 b, and a bandwidth parameter76 c, 77 c, 78 c, 79 c, 80 c, 81 c. The high shelf equalizer 82 caninclude a frequency parameter 82 a, a gain parameter 82 b, and abandwidth parameter 82 c. The resulting outputs from the low-shelfequalizer 75, the band-boost/cut equalizers 76, 77, 78, 89, 80, 81, andthe high shelf equalizer 82 are summed 84 a, 84 b, producing theproximity effect module output signal 85.

FIGS. 23A and 23B illustrates an eight-band equalizer an example of aconfiguration that could be used emulate proximity effect. Otherconfigurations are possible. For example, as few as a single-bandequalizer centered about the proximity boost frequency could simulatethe proximity effect boost. As few as three-band equalizer: onehigh-frequency shelving, low-frequency shelving, and one mid-bandcentered about the proximity-effect boost frequency could simulate highfrequency roll-off, low frequency roll-off, and proximity effect. Afour-band equalizer: one high-frequency shelving, low-frequencyshelving, and one mid-band centered about the proximity-effect boostfrequency (typically about 200 Hz.), one mid-band centered about thepresence rise frequency (typically about 5 kHz.) could emulate highfrequency roll-off, low frequency roll-off, proximity effect, and thehigh-frequency peak common in vocal microphones known as presence. Notethat with the careful use of bandwidth or Q with the high-shelfequalizer, a three-band equalizer can also low-frequency roll-off,high-frequency roll-off, proximity effect, and presence.

FIG. 24 illustrates a simplified block diagram of the feedback effectemulation module 42 of FIGS. 18 and 19 that produces simulatedmicrophone feedback. Referring to FIG. 24, based on proximity data fromthe control signal 11, a range module 91 compares the proximity datafrom the control signal 11 to a pre-determined distance range. Thepre-determined distance range is typically determined by the controlsignal 62 by the user parameter controls 22 from FIGS. 18 and 19. If theproximity data is within the pre-determined range the feedback soundgenerator 92 initializes the feedback sequence 92 a, selects the filevia a file selector module 92 b, selects 93 and plays a pre-recordedfeedback sound sample 92 c from the pre-recorded feedback samples folder94. The output signal level of feedback sound generator 92 is scaled bya level control 95. A data scaling module 96 uses the proximity datafrom the control signal 11 to control the level control 95 and scale thefeedback output 97. The file selector module 92 b can use input from thecharacter control 73 c or probability control 73 b of FIG. 22 viacontrol signal 62 from the user parameter controls 22 of FIGS. 18 and19. The file selector could also use a random or pseudo-random numbergenerator to select the pre-recorded feedback sample from thepre-recorded feedback samples folder 94.

FIG. 25 illustrates a simplified block diagram of a typicalimplementation of the distortion effect emulation module 41 illustratedin FIG. 18 or 19. Referring to FIG. 25, the distortion effect can becreated by feeding the audio data stream 8 from the pre-recorded audiobacking track into a multi-band audio filter 100. In FIG. 25, themulti-band audio filter 100 is illustrated as a three-band filter with alow-pass filter 101, band-pass filter 102, and high-pass filter 103. Theoutputs of the low-pass filter 101, band-pass filter 102, and high-passfilter 103 feed the wave-shaping synthesizers 104, 105, 106,respectively. The wave-shaping synthesizers 104, 105, 106 creates thedistortion effect by adding harmonics to the low-pass filter 101,band-pass filter 102, and high-pass filter 103 outputs. The output ofthe wave-shaping synthesizers 104, 105, 106 are summed 107 to create thedistortion effect output 108. The low-pass filter 101, band-pass filter102, and high-pass filter 103 each include a gain adjustment 101 a, 102a, 103 a and frequency adjustment 101 b, 102 b, 103 b, respectively. Acontrol signal 109 a from a data scaling module 109 adjusts the gain ofeach of the filter components based on data it receives from the controlsignal 11 based on the dynamic distance data stream 12 of FIG. 18 or 19.The frequency adjustments 101 b, 102 b, 103 b and the wave-shapingsynthesizers 104, 105, 106 can optionally be controlled by the controlsignal 62 from the user parameter controls 22 of FIGS. 18 and 19.

FIGS. 26-29 illustrate different ways the proximity sensor assembly 7can be mounted a fixed distance from the end 2 a of the microphonegrille 2 b. In FIG. 26 the stage microphone 2 is mounted to a microphonestand 4 and the proximity sensor assembly 7 is mounted on the microphonebody 2 c. The proximity sensor assembly 7 can include an enclosure 7 bthat houses the proximity sensor 7 a, a circuit board to hold circuitryassociated with the proximity sensor 7 a, and a battery for supplyingpower to the proximity sensor and circuit board. The enclosure 7 b canbe secured to the microphone body 2 c by any structure that allows theenclosure 7 b to stay secured to the microphone body 2 c under thenormal operation of a live musical performance. For example, theenclosure 7 b can be directly secured by adhesive or an adhesive stripaffixed to the underside of the proximity sensor assembly 7. Theenclosure 7 b can be flanged with the flange screwed into the microphonebody 2 c. The enclosure 7 b can be secured to the microphone body 2 c bya tension band. The proximity sensor 7 a is shown as wireless and notrequiring a cable. The stage microphone 2 is shown as a wired microphonerequiring a cable 110 for routing the audio signal. One advantage ofthis configuration is that the vocalist or performer can remove thestage microphone 2 from the microphone stand 4 during a liveperformance, but still be able to take advantage of proximity dependentlip-synchronized level and effects previously described because theproximity sensor 7 a remains attached a fixed distance from the end 2 aof the microphone grille 2 b and therefore the distance to theperformer's face or lips can be measured.

In FIG. 27 the enclosure 7 b of the proximity sensor assembly 7 ismounted to a microphone clip 4 a that attaches stage microphone 2 to themicrophone stand 4. Here the proximity sensor 7 a and the stagemicrophone 2 are both shown as wired, and requiring the cables 110, 111to transmit both the audio signal and the dynamic distance data stream12 of FIGS. 7, 18, and 19. The enclosure 7 b can be secured to themicrophone clip 4 a by any structure that allows the enclosure 7 b tostay secured to the microphone clip 4 a under the normal operation of alive musical performance. For example, the enclosure 7 b can be directlysecured by adhesive or an adhesive strip affixed to the underside of theproximity sensor assembly 7. The enclosure 7 b can be flanged with theflange screwed into the microphone clip 4 a. This arrangement requiresthat the stage microphone 2 remain mounted to the microphone stand 4since the proximity sensor assembly 7 is secured to the microphone clip4 a. One advantage of this arrangement is that proximity sensor 7 a isnot as shadowed or potentially blocked by the microphone windscreen asin the example shown in FIG. 26.

FIG. 28 illustrates a stage microphone 2 and microphone stand 4 wherethe proximity sensor assembly 7 is mounted partially within the body ofthe microphone with the proximity sensor 7 a extending out of themicrophone body 2 c and upward around the bottom half of the microphonegrille 2 b. As with the example of FIG. 26, this configuration can beadvantageous when the vocalist or performer uses the microphone with andwithout the microphone stand 4 during a performance because theproximity sensor assembly 7 is secured directly to the stage microphone2 and therefore maintains a constant and known distance from the end 2 aof the microphone grille 2 b. This configuration also avoids anyshadowing or blocking of the proximity sensor 7 a by the microphonegrille 2 b.

FIG. 29 illustrates a partially exploded view of the stage microphone 2and proximity sensor assembly 7 of FIG. 28. The microphone grille 2 b isexploded away revealing the microphone capsule 2 e and the proximitysensor assembly 7 including elements hidden from view in FIG. 28. Hiddenelements are illustrated as dashed lines. Referring to FIG. 29, theproximity sensor 7 a is mounted at the end of a mounting arm 7 c. Themounting arm 7 c can be made of a material that is flexible but able tohold its shape. For example, the mounting arm could be made from hollowflexible conduit. This would allow electric wires to pass through themounting arm 7 c to the printed circuit board 7 d that is housed withinthe microphone body 2 c. This also allows the mounting arm 7 c to beshaped by the sound engineer to conform to the outside of the microphonegrille 2 b. The mounting arm 7 c includes a ring-shaped base 7 e thatseats on the windscreen mounting flange 2 f. The printed circuit board 7d includes circuitry associated with the proximity sensor 7 a; forexample, a battery 7 f or integrated circuits 7 g. The integratedcircuits 7 g could include a microcontroller, field programable gatearray (FPGA), a programable logic device (PLD), or a function-specificintegrated circuit. Examples of function-specific integrated circuitsinclude a I2C or I2S to USB converter or an I2C or I2S to wirelessconverter. The output of the printed circuit board 7 d can optionallyinclude proximity module output wires 7 h that feed a microphone outputconnector 2 g. For example, the microphone output connector 2 g could bea five-pin XLR connector. A five-pin XLR connector can accommodate theinternal microphone cable 2 h and the proximity module output wires 7 h.This configuration may be desirable where only the proximity data isbeing transmitted from the microphone and not a live microphone signal.If a live microphone signal is also transmitted, care should be taken toavoid cross-talk between the proximity data and the audio signal since amicrophone signal typically has a signal level in the millivolt rangeand may be susceptible to cross-talk.

FIGS. 30-32 illustrate examples of simplified electrical circuitdiagrams for the proximity sensor assembly 7. Referring to FIGS. 30-32,the proximity sensor 7 a is typically powered by an internal powersupply such as the battery 7 f and voltage regulator 7 j illustrated.The proximity data can be transmitted by USB or another wired computerprotocol. For example, in FIG. 30 the proximity data from the proximitysensor 7 a is illustrated as I2C serial data. This can be converted toUSB by an I2C to USB data via a FPGA, microprocessor, PLD, or adedicated device such as the illustrated I2C to USB serial converter 7k. Examples of I2C to USB serial converters include the MCP2221A fromMicrochip Technology, Inc. or a TUSB3410 from Texas InstrumentsIncorporated. The proximity sensor data can also be transmittedwirelessly. For example, in FIG. 31, I2C data from the proximity sensor7 a is converted to wireless data via a microcontroller or a wirelessdata converter 7 n, such as 802.11 and transmitted via an antenna 7 m.An example of a wireless data converter 7 n is the CYW43903 by CypressSemiconductor. It may be desirable to convert the proximity sensor datadirectly to a standard audio control protocol such as MIDI or OSC. FIG.32 illustrates serial data from the proximity sensor 7 a feeding amicrocontroller 7 p. The microcontroller includes instructions stored inmemory such as internal or external FLASH memory that when executedcause the microcontroller to convert the serial data from the proximitysensor 7 a to MIDI or OSC. FIG. 32 also illustrates an example of asimplified electrical diagram for FIG. 29 where the output of themicrophone capsule 2 e and the output of the microcontroller 7 p canboth feed the microphone output connector 2 g.

FIGS. 33-35 illustrate several examples of how the distance-appliedlevel and effects emulation system can be applied in a live soundenvironment, which typically includes an audio mixing console 113 (i.e.,a live sound mixing console or front of house mixing console), audioamplifiers 114, and speakers 115. In each of the examples, the vocalist117 is positioned a distance D from the proximity sensor 7 a of theproximity sensor assembly 7. The proximity sensor assembly 7 is mountedin a fixed distance relationship with stage microphone 2. For example,the proximity sensor assembly can be affixed to the stage microphone 2or the microphone clip as previously described. FIG. 33 illustrates asimplified system block diagram 112 where distance-applied level andeffects emulation is processed within a digital audio workstation 118 asa plugin 118 a. The digital audio workstation 118 can be a standaloneunit or can be hosted on a computer 119 as illustrated. Examples ofdigital audio workstations 118 running hosted on a computer 119 are ProTools by Avid Technology Inc., Ableton Live by Ableton AG, and DigitalPerformer by Mark of the Unicorn. The plugin 118 a can be written in astandard or proprietary format. Standard formats include Virtual StudioTechnology (VST), Audio Units, and TDM. The dynamic distance data stream12, or alternatively the control signal 11, from the proximity sensor 7a can be transmitted to the computer 119 hosting the digital audioworkstation 118 by wire or wirelessly; for example, by USB or 802.11.The audio output 57 feeds an input of the audio mixing console 113. Thiscan be either an analog signal or a digital audio signal such AES/EBU.Optionally, a live audio signal 120 can feed an input channel of theaudio mixing console 113 so the vocal performance can be mixed with thepre-recorded audio backing track as previously discussed.

Referring to FIG. 34, the distance-applied level and effects emulationis processed within a standalone unit 124. The standalone unit mayinclude the pre-recorded audio backing tracks 9 of FIGS. 7, 18, and 19in internal storage or use an external storage unit such as a harddrive, flash memory, or a digital audio recorder 125, as illustrated.The standalone unit 124 can receive the control signal 11, such as MIDI,or OSC, directly processed by proximity sensor assembly 7.Alternatively, the standalone unit 124 can receive the dynamic distancedata stream 12 from the proximity sensor assembly 7 and convert it toMIDI, OSC, or other standard control signal protocol. The audio output57 representing the audio data stream 8 of the pre-recorded audiobacking track 9 of FIGS. 7, 18, and 19, with distance-applied level andeffects emulation, feeds an input channel of the audio mixing console113. The live audio signal 120, if required, can be optionally fed fromthe stage microphone 2 to the audio mixing console 113. The standaloneunit 124 can include an integrated digital audio mixer. In that case,the live audio signal 120 can optionally be pre-mixed and the combinedsingle sent to the audio mixing console 113.

Referring to FIG. 35, in the case of a wireless microphone, for example,the stage microphone 2 of FIGS. 1 and 2, the dynamic distance datastream 12 (or control signal 11) can be transmitted together with awireless audio signal to a wireless microphone receiver 121. The liveaudio signal 120, if required can be optionally fed from the wirelessmicrophone receiver to the audio mixing console 113. The audio output 57from the digital audio workstation 118, that represents thedistance-adjusted pre-recorded audio backing track, feeds an inputchannel on the audio mixing console 113. The dynamic distance datastream 12 can be fed from the wireless microphone receiver 121 to thecomputer 119 hosting the digital audio workstation 118 and plugin 118 a.The wireless microphone receiver 121 can optionally include the controlsignal protocol conversion module of FIGS. 7, 18, and 19. In that case,the control signal 11, that results from the dynamic distance datastream 12, can be fed to the computer 119, as illustrated. If thedigital audio workstation is a standalone device, the dynamic distancedata stream 12 can be sent directly from the wireless microphonereceiver 121 to the digital audio workstation 118.

There may be circumstances where the performance can be enhanced byhaving the vocalist or other performer have some control over thedistortion, feedback, or proximity effect. For example, the vocalist 3of FIGS. 1 and 2 may purposely want to over exaggerate proximity effectas they move the stage microphone 2 close to their lips. FIG. 36illustrates a block diagram of a distance-applied level and effectsemulation system with user controls 127 on the stage microphone. FIGS.37 and 38 illustrate two examples of user controls 127 mounted on themicrophone body 2 c of a stage microphone 2. Referring to FIGS. 37 and38, the user controls 127 can be molded into or around the microphonebody 2 c as illustrated. The user controls 127 can include switchesunder a rubberized protective covering. Alternatively, the user controls127 can use a force sensor or force sensors to control effect intensity;for example, to control the intensity of feedback or proximity effect.Suitable force sensors could include force sensing resistors orpiezoelectric forces sensors. FIG. 37 illustrates a user control 127surrounding the circumference of the microphone body 2 c. FIG. 38illustrates the user controls as having control zones 127 a, 127 b, 127c. These three control zones can be used to control different effects,for example proximity effect, feedback, or distortion. They couldalternatively be used to control different parameters of a singleeffect. For example, the peak frequency and intensity of a proximityeffect. The illustration of user control 127 in FIG. 37 or the usercontrol 127 with three of the control zones 127 a, 127 b, 127 c in FIG.38 are examples of what is possible other combinations of control zonesand user controls mounted to a microphone are possible. In addition,while the user control 127 in FIGS. 37 and 38 are illustrated assurrounding the circumference of the microphone body 2 c, otherconfigurations are possible, for example a simple switch secured withinor on the surface of the microphone body 2 c.

Referring to FIG. 36, the system diagram is nearly identical to thesystem diagram of FIG. 19 except for the addition of user controls 127,user control signal processing module 128, and the control signal 129.The proximity sensor 7 a, the audio data stream 8, the pre-recordedaudio backing track 9, the control signal 11, the control signalprotocol conversion module 13, the level-versus-distance emulationmodule 10, the user parameter controls 22, the proximity effectemulation module 33, the distortion effect emulation module 41, thefeedback effect emulation module 42, level control modules 52, 53, 54,55, the summer 56, and the audio output 57 are as described for FIG. 19.The performer or vocalist activates the user controls 127 on themicrophone body 2 c of FIGS. 38 and 39. The user control data from theuser controls 127 is converted into a usable control format via the usercontrol signal processing module 128. If the user control data from theuser controls 127 is already in a usable control format, this module canbe eliminated. The control signal 129 can feed the proximity effectemulation module, the distortion effect emulation module, the feedbackeffect emulation module, or the level control modules 52, 53, 54, 55 inorder to control one or more user parameters, as described above. Thesystem diagram of FIG. 36 illustrates how the user controls 127 can beapplied to the system diagram of FIG. 19. In a similar manner, the usercontrols 127 can be applied to the system diagram of FIGS. 7 and 18.

The inventors envision the following additional embodiments, labeledbelow as examples, are also within the scope of the distance-appliedlevel and effects emulation described within this disclosure.

Example 1

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer;

a non-transitory computer readable medium including instructions storedtherein that when executed by a processor cause the processor todynamically modify an audio data stream of a pre-recorded audio backingtrack in response to the dynamic distance data stream.

Example 2

The system of Example 1, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a signal level of the audiodata stream inversely with the distance in response to the dynamicdistance data stream.

Example 3

The system of Example 1, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a signal level of the audiodata stream in such a way that emulates distance-dependent changes inmicrophone level.

Example 4

The system of Example 1, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a low frequency response ofthe audio data stream inversely with the distance, in response to thedynamic distance data stream.

Example 5

The system of Example 1, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a frequency response of theaudio data stream in response to the dynamic distance data stream insuch a way that emulates distance-dependent proximity effect.

Example 6

The system of Example 1, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically add distortion to the audio datastream in response to the dynamic distance data stream in such a waythat emulates distance-dependent microphone distortion.

Example 7

The system of Example 1, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically add simulated microphone feedback tothe audio data stream in response to the dynamic distance data stream insuch a way that emulates distance-dependent microphone feedback.

Example 8

A method for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

receiving a dynamic distance data stream generated by a proximitysensor, the proximity sensor secured in a fixed relationship with thestage microphone, the dynamic distance data stream representing adistance between the proximity sensor and the performer; anddynamically modifying an audio data stream of a pre-recorded audiobacking track in response to the dynamic distance data stream.

Example 9

The method of Example 8 further comprising:

adjusting a signal level of the audio data stream inversely with thedistance in response to the dynamic distance data stream.

Example 10

The method of Example 8 further comprising:

dynamically adjusting a signal level of the audio data stream in such away that emulates distance-dependent changes in microphone level.

Example 11

The method of Example 8 further comprising:

dynamically adjust a low frequency response of the audio data streaminversely with the distance in response to the dynamic distance datastream.

Example 12

The method of Example 8 further comprising:

dynamically adjusting a frequency response of the audio data stream inresponse to the dynamic distance data stream in such a way that emulatesdistance-dependent proximity effect.

Example 13

The method of Example 8 further comprising:

dynamically add distortion to the audio data stream in response to thedynamic distance data stream in such a way that emulatesdistance-dependent microphone distortion.

Example 14

The method of Example 8 further comprising:

dynamically adding simulated microphone feedback to the audio datastream in response to the dynamic distance data stream in such a waythat emulates distance-dependent microphone feedback.

Example 15

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer;

a processor;

a memory to store computer-executable instructions that, if executed,cause the processor to dynamically modify an audio data stream of apre-recorded audio backing track in response to the dynamic distancedata stream.

Example 16

The system of Example 15, wherein:

the processor dynamically adjusts a signal level of the audio datastream inversely with the distance in response to the dynamic distancedata stream.

Example 17

The system of Example 15, wherein:

the processor dynamically adjusts a signal level of the audio datastream in such a way that emulates distance-dependent changes inmicrophone level.

Example 18

The system of Example 15, wherein:

the processor to dynamically adjusts a low frequency response the audiodata stream inversely with the distance, in response to the dynamicdistance data stream.

Example 19

The system of Example 15, wherein:

the processor dynamically adjusts a frequency response of the audio datastream in response to the dynamic distance data stream in such a waythat emulates distance-dependent proximity effect.

Example 20

The system of Example 15, wherein:

the processor dynamically adds distortion to the audio data stream inresponse to the dynamic distance data stream in such a way that emulatesdistance-dependent microphone distortion.

Example 21

The system of Example 15, wherein:

the processor dynamically adds simulated microphone feedback to theaudio data stream in response to the dynamic distance data stream insuch a way that emulates distance-dependent microphone feedback.

Example 22

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and

a level-versus-distance emulation module dynamically adjusts a signallevel of an audio data stream of a pre-recorded audio backing track inresponse to the dynamic distance data stream.

Example 23

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and

a level-versus-distance emulation module dynamically adjusts a signallevel of an audio data stream of a pre-recorded audio backing track inresponse to the dynamic distance data stream in such a way that emulatesdistance-dependent changes in microphone level.

Example 24

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and

a proximity effect emulation module dynamically adjusts a low frequencyresponse of an audio data stream of a pre-recorded audio backing trackinversely with the distance in response to the dynamic distance datastream.

Example 25

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and

a proximity effect emulation module dynamically adjusts a frequencyresponse of a pre-recorded audio backing track in response to thedynamic distance data stream in such a way that emulatesdistance-dependent proximity effect.

Example 26

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and

a distortion effect emulation module dynamically adds distortion toaudio data stream of a pre-recorded audio backing track in response tothe dynamic distance data stream in such a way that emulatesdistance-dependent microphone distortion.

Example 27

A system for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising:

a proximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and

a feedback effect emulation module dynamically adds simulated microphonefeedback to audio data stream of a pre-recorded audio backing track inresponse to the dynamic distance data stream in such a way that emulatesdistance-dependent microphone feedback.

Example 28

A system of Example 27, wherein:

the feedback effect emulation module includes a feedback soundgenerator, a range module, and a pre-recorded feedback sample;

the range module in response to the dynamic distance data stream causesthe feedback sound generator to retrieve and play the pre-recordedfeedback sample when the dynamic distance data stream is within apre-determined range.

Example 29

A system of Example 27, wherein:

the feedback effect emulation module in response to the dynamic distancedata stream plays a pre-recorded feedback sample when the dynamicdistance data stream is within a pre-determined range.

Example 30

A stage microphone for enhancing a lip-synchronized performance of aperformer, including:

a proximity sensor secured to the stage microphone;

the proximity sensor generates a dynamic distance data streamdynamically representing a distance between the proximity sensor and theperformer; and

the stage microphone produces a standard audio control protocol thatincludes a dynamic distance data from the dynamic distance data stream.

Example 31

The stage microphone of Example 30, further including:

the stage microphone includes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 32

The stage microphone of Example 31, wherein the user control includes aforce sensor.

Example 33

A system for enhancing a lip-synchronized performance of a performer,comprising:

a stage microphone;

a proximity sensor secured to the stage microphone;

the proximity sensor generates a dynamic distance data streamdynamically representing a distance between the proximity sensor and theperformer;

the stage microphone produces a standard audio control protocol thatincludes a dynamic distance data from the dynamic distance data stream;

a non-transitory computer readable medium including instructions storedtherein that when executed by a processor cause the processor todynamically modify an audio data stream of a pre-recorded audio backingtrack in response to the dynamic distance data stream.

Example 34

The system of Example 33, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a signal level of the audiodata stream inversely with the distance in response to the dynamicdistance data stream.

Example 35

The system of Example 33, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a signal level of the audiodata stream in such a way that emulates distance-dependent changes inmicrophone level.

Example 36

The system of Example 33, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a low frequency response ofthe audio data stream inversely with the distance, in response to thedynamic distance data stream.

Example 37

The system of Example 33, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically adjust a frequency response of theaudio data stream in response to the dynamic distance data stream insuch a way that emulates a distance-dependent proximity effect.

Example 38

The system of Example 33, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically add a distortion to the audio datastream in response to the dynamic distance data stream in such a waythat emulates distance-dependent microphone distortion.

Example 39

The system of Example 33, wherein:

the instructions stored in the non-transitory computer readable mediumcause the processor to dynamically add a simulated microphone feedbackto the audio data stream in response to the dynamic distance data streamin such a way that emulates distance-dependent microphone feedback.

Example 40

The system of Example 33, further including: the stage microphoneincludes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 41

The system of Example 40, wherein the user control data modifies theaudio data stream.

Example 42

The system of Example 41, wherein the user control includes a forcesensor.

Example 43

The system of Example 42, wherein the user control data modifies theaudio data stream in response to a force sensed from the user control.

Example 44

The system of Examples 34 or 35, further including:

the stage microphone includes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 45

The system of Example 44, wherein the user control data modifies thesignal level of the audio data stream.

Example 46

The system of Example 44, wherein the user control includes a forcesensor.

Example 47

The system of Example 46, wherein the user control data modifies thesignal level of the audio data stream in response to a force sensed fromthe user control.

Example 48

The system of Example 37, further including:

the stage microphone includes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 49

The system of Example 48, wherein the user control data modifies thefrequency response.

Example 50

The system of Example 48, wherein the user control includes a forcesensor.

Example 51

The system of Example 48, wherein the user control data modifies thefrequency response in response to a force sensed from the user control.

Example 52

The system of Example 38, further including: the stage microphoneincludes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 53

The system of Example 52, wherein the user control data modifies thedistortion.

Example 54

The system of Example 52, wherein the user control includes a forcesensor.

Example 55

The system of Example 52, wherein the user control data modifies thedistortion in response to a force sensed from the user control.

Example 56

The system of Example 38, further including:

the stage microphone includes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 57

The system of Example 56, wherein the user control data modifies thedistortion.

Example 58

The system of Example 39, further including:

the stage microphone includes a microphone body;

a user control attached to the microphone body, the user control outputsa user control data; and

the standard audio control protocol includes the user control data.

Example 59

The system of Example 58, wherein the user control data modifies thesimulated microphone feedback.

Example 60

The system of Example 58, wherein the user control includes a forcesensor.

Example 61

The system of Example 58, wherein the user control data modifies thesimulated microphone feedback in response to a force sensed from theuser control.

Example 62

The system of Examples 2 or 3, further comprising:

a user parameter controls; and

the user parameter controls adjust a scale factor of the signal level ofthe audio data stream.

Example 63

The system of Examples 2 or 3, further comprising:

a user parameter controls;

the user parameter controls adjust a level-versus-distance law of thesignal level of the audio data stream.

The preceding figures and specification describe a system and method forcreating more realistic lip syncing to pre-recorded vocal and acousticinstrument tracks during live performances (i.e. a system for enhancinga lip synchronized performance or a method for enhancing a lipsynchronized performance). It is not the intent of this disclosure tolimit the claimed invention to the examples, variations, and exemplaryembodiments described in the specification. Those skilled in the artwill recognize that variations will occur when embodying the claimedinvention in specific implementations and environments. For example, thesystem of FIGS. 1-38 were discussed in the context of a live soundenvironment. These include, for example, a live concert in a stadium,amphitheater, concert hall, cruise ship, night club, or live televisionperformance. The inventor envisions that the described system might alsobe useful in a broad range of applications that can benefit fromrealistic lip sync performances. For example, the system can also beutilized for film, pre-recorded television, or music video recording.For example, in film, pre-recorded television, or music videoproduction, the vocalist or musicians typically lip syncs to apre-recorded audio backing track while they stand near a microphone propor while the vocalist holds a microphone prop. The dynamic distance datastream 12 from the proximity sensor 7 a of FIG. 7, 18, or 19, forexample, can be used during sound editing or post production to createmore realistic lip syncing.

In FIGS. 30-32, the data transmitted from the proximity sensor 7 a wasillustrated as I2C. While some proximity sensors, such as the VL53L0Xtime-of-flight proximity sensor by ST Microelectronics or the OPT3101from Texas Instruments output their proximity data via an I2C serialport, other time-of-flight sensors have other output data ports that oneof ordinary skill in the art would readily be able to use without undueexperimentation. In FIG. 30, the output is illustrated as USB. Theoutput could also be other wired computer protocols such as Ethernet. InFIG. 31, an 802.11 output was discussed. However, other wirelessprotocols could readily be used such as Zigbee or BlueTooth.

It is possible to implement certain features described in separateembodiments in combination within a single embodiment. Similarly, it ispossible to implement certain features described in single embodimentseither separately or in combination in multiple embodiments. Forexample, FIGS. 1 and 2 illustrate a wireless microphone and FIGS. 3 and4 illustrate a wired microphone. The proximity sensor assembly 7 isillustrated in at least three different mounting configurations; forexample, the mounting configurations of FIG. 26, 27, or 28. Thesemounting configurations can be used with stage microphone 2 that areboth wired or wireless. In FIGS. 30 and 32, the output of the proximitysensor assembly 7 is illustrated as wired. In FIG. 31, the output of theproximity sensor assembly is illustrated as wireless. Wired or wirelessconfigurations can be used for any of the examples of FIGS. 1-6 and canbe applied to the simplified block diagrams of FIGS. 7, 18, 19, and 37.The inventor envisions that these variations fall within the scope ofthe claimed invention.

Any appended claims are not to be interpreted as includingmeans-plus-function limitations, unless a claim explicitly evokes themeans-plus-function clause of 35 USC § 112(f) by using the phrase “meansfor” followed by a verb in gerund form.

“Optional” or “optionally” is used throughout this disclosure todescribe features or structures that are optional. Not using the wordoptional or optionally to describe a feature or structure does not implythat the feature or structure is essential, necessary, or not optional.Using the word “or,” as used in this disclosure is to be interpreted asthe ordinary meaning of the word “or” (i.e., an inclusive or) Forexample, the phrase “A or B” can mean: (1) A, (2) B, (3) A with B. Forexample, if one were to say, “I will wear a waterproof jacket if itsnows or rains,” the meaning is that the person saying the phraseintends to wear a waterproof jacket if it rains alone, if it snowsalone, if it rains and snows in combination.

While the examples, exemplary embodiments, and variations are helpful tothose skilled in the art in understanding the claimed invention, itshould be understood that, the scope of the claimed invention is definedsolely by the following claims and their equivalents.

What is claimed is:
 1. A system for enhancing a lip-synchronizedperformance of a performer utilizing a stage microphone, comprising: aproximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and a non-transitory computer readable mediumincluding instructions stored therein that when executed by a processorcause the processor to dynamically modify a signal level of an audiodata stream of a pre-recorded audio backing track in response to thedynamic distance data stream inversely with the distance between theproximity sensor and the performer.
 2. A system for enhancing alip-synchronized performance of a performer utilizing a stagemicrophone, comprising: a proximity sensor secured in a fixedrelationship with the stage microphone, the proximity sensor generatinga dynamic distance data stream dynamically representing a distancebetween the proximity sensor and the performer; and a non-transitorycomputer readable medium including instructions stored therein that whenexecuted by a processor cause the processor to dynamically modify a lowfrequency response of an audio data stream of a pre-recorded audiobacking track in response to the dynamic distance data stream inverselywith the distance between the proximity sensor and the performer.
 3. Amethod for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising: receiving a dynamic distancedata stream generated by a proximity sensor, the proximity sensorsecured in a fixed relationship with the stage microphone, the dynamicdistance data stream representing a distance between the proximitysensor and the performer; and dynamically modifying a signal level of anaudio data stream of a pre-recorded audio backing track in response tothe dynamic distance data stream inversely with the distance between theproximity sensor and the performer.
 4. A method for enhancing alip-synchronized performance of a performer utilizing a stagemicrophone, comprising: receiving a dynamic distance data streamgenerated by a proximity sensor, the proximity sensor secured in a fixedrelationship with the stage microphone, the dynamic distance data streamrepresenting a distance between the proximity sensor and the performer;and dynamically modifying a low frequency response of an audio datastream of a pre-recorded audio backing track in response to the dynamicdistance data stream inversely with the distance between the proximitysensor and the performer.
 5. A system for enhancing a lip-synchronizedperformance of a performer utilizing a stage microphone, comprising: aproximity sensor secured in a fixed relationship with the stagemicrophone, the proximity sensor generating a dynamic distance datastream dynamically representing a distance between the proximity sensorand the performer; and a processor that in response to the dynamicdistance data stream dynamically modifies a signal level of an audiodata stream of a pre-recorded audio backing track inversely with thedistance between the proximity sensor and the performer.
 6. A system forenhancing a lip-synchronized performance of a performer utilizing astage microphone, comprising: a proximity sensor secured in a fixedrelationship with the stage microphone, the proximity sensor generatinga dynamic distance data stream dynamically representing a distancebetween the proximity sensor and the performer; a processor that inresponse to the dynamic distance data stream dynamically modifies a lowfrequency response of an audio data stream of a pre-recorded audiobacking inversely with the distance between the proximity sensor and theperformer.
 7. A system for enhancing a lip-synchronized performance of aperformer utilizing a stage microphone, comprising: a proximity sensorsecured in a fixed relationship with the stage microphone, the proximitysensor generating a dynamic distance data stream dynamicallyrepresenting a distance between the proximity sensor and the performer;a non-transitory computer readable medium including instructions storedtherein that when executed by a processor cause the processor todynamically modify an audio data stream of a pre-recorded audio backingtrack in response to the dynamic distance data stream in such a way thatsimulates changes in microphone signal level, proximity effect,distortion effect, and/or feedback effect in relation to correspondingchanges in the distance between the proximity sensor and the performer.8. A method for enhancing a lip-synchronized performance of a performerutilizing a stage microphone, comprising: receiving a dynamic distancedata stream generated by a proximity sensor, the proximity sensorsecured in a fixed relationship with the stage microphone, the dynamicdistance data stream representing a distance between the proximitysensor and the performer; and dynamically modifying an audio data streamof a pre-recorded audio backing track in response to the dynamicdistance data stream in such a way that simulates changes in microphonesignal level, proximity effect, distortion effect, and/or feedbackeffect in relation to corresponding changes in the distance between theproximity sensor and the performer.
 9. A system for enhancing alip-synchronized performance of a performer utilizing a stagemicrophone, comprising: a proximity sensor secured in a fixedrelationship with the stage microphone, the proximity sensor generatinga dynamic distance data stream dynamically representing a distancebetween the proximity sensor and the performer; a processor that inresponse to the dynamic distance data stream dynamically modifies anaudio data stream of a pre-recorded audio backing track in such a waythat simulates changes in microphone signal level, proximity effect,distortion effect, and/or feedback effect in relation to correspondingchanges in the distance between the proximity sensor and the performer.